Commentary: Continuously cumulating meta-analysis and replicability

Citation: Perezgonzalez JD (2015) Commentary: Continuously cumulating meta-analysis and replicability. Front. Psychol. 6:565. A commentary on Continuously cumulating meta-analysis and replicability Braver et al. (2014) article was published by Perspectives on Psychological Science as part of a special issue on advancing psychology toward a cumulative science. The article contributes to such advance by proposing using meta-analysis cumulatively, rather than waiting for a long number of replications before running such meta-analysis. Braver et al.'s article sits well alongside a recent call for reforming psychological methods, under the umbrella of " the new statistics " (Cumming, 2012). As it happens with the latter, the method referred to is not new, only the call to use it is. Indeed, the idea behind a continuously cumulating meta-analysis (CCMA) was already put forward by Rosenthal as far back as 1978 and repeated since (e.g., Rosenthal, 1984, 1991). Yet, the reminder is as relevant today as it has been in the past, more so if we want to get psychology, and our own research within it, at the frontier of science. I will, however, take this opportunity to comment on an issue which I find contentious: the meaning of the replication used to prove the point. Braver et al. define the criterion for a successful replication as achieving conventional levels of significance. They also identify the typical low power of psychological research as a main culprit for failing to replicate studies. Indeed, they went ahead and simulated two normal populations with a medium effect size mean difference between them, from which they randomly drew 10,000 pairs of underpowered samples. The results they obtained fulfilled power expectations: about 42% of the initial studies, about 41% of the replications, and about 70% of the combined study-replication pairs turned out statistically significant—the latter supposedly supporting the benefits of CCMA over the uncombined studies. What the authors fail to notice, however, is that the meaning of replication differs depending on the data testing approach used: Fisher's approach is not the same than Neyman–Pearson' Perezgonzalez, 2015a). Neyman and Pearson's approach (1933) is based on repeated sampling from the same population while keeping an eye on power, which is Braver et al.'s simulation setting (Neyman and Pearson, 1933). However, under this approach a successful replication reduces to a count of significant results in the long run, which translates to about 80% of significant replications when power is 0.8, or to about 41% …